Embedded Deep Learning Multi-Scale Object Detection Model Using Real-Time Distant Region Locating Device and Method Thereof

ABSTRACT

The present invention provides a deep learning object detection method that locates the distant region in the image in real-time and concentrates on distant objects in a front dash cam perspective, trying to solve a common problem in advanced driver assistance system (ADAS) applications, that the detectable range of the system is not far enough.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an object detection field, particularly to an embedded deep learning multi-scale object detection model using real-time distant region locating device and method thereof.

2. Description of the Prior Art

Recently, in the computer information field, due to the flourishing development of deep learning techniques, the expression of computer vision technique has already been approaching the expression of human eyes, and the relevant technical topics have become very hot correspondingly. In these computer vision techniques, a large amount of computer vision understanding techniques have been used in the advanced driver assistance system (ADAS), where the object detection techniques have been used in various car safety assist driving systems, such as Forward Collision Warning System (FCWS), Rear Collision Warning System (RCWS), Brake Assist System (BAS), Advanced Emergency Braking System (AEBS), Lane Departure Warming System (LDWS), and Blind Spot Detection System (BSDS), etc. At present, the above-mentioned systems can be effectively used in automatic assist car driving, which not only can effectively lighten the long driving fatigue, but also can effectively achieve the car driving assist effect, and further increase the safety of driving.

When the car is driven at high speed, the detection of distant object will become very important. However, the distant object will usually be shown as a small object in the detected image, also, the characteristics will be presented less. Based on the imaged object recognized under distant object detection, the obtained effect will be poorer. Thus, a lot of traffic accidents occur because it is unable to dodge the object suddenly resting on the lane in time, such as the stalled vehicle, vehicle stayed in front of road due to traffic accident, or other obstacles rested in front of road. Therefore, detecting the small distant objects at this moment has become an important and necessary detection technique for ensuring safe car driving.

As for the conventional detection of small distant objects, it is necessary to adopt the high-resolution image for regarding as the object recognition, in order to carry on follow-up analysis for obtaining enough object characteristics. However, it will take a large amount of computation time by directly using high-resolution image to catch the characteristics. If the whole picture is caught directly, it will cause that the characteristics of small objects will be insufficient to detect sufficient information. Most conventional objects detection systems are rule-based techniques. It is necessary to aim at various object characteristics and use specific characteristics catching methods to define the objects, such as the car, people, etc. If there is abominable weather or complicated environmental background, or the uncommon object appeared on the road, or the characteristics of object are lost due to deformation or fast movement of object, it will cause substantial drop of detection accuracy and stability. Thus, as for the conventional image detection technique, such as S. Teoh's research team uses the image edge detection method and the left-right symmetrical characteristics of car, to generate the probable appearance position of detected target object. However, if the car direction taken by camera is not located right behind the car, the car will not possess the symmetrical characteristics, so that the detection effect will be influenced tremendously. The detected object cannot be too small, it is necessary to possess certain amount of edge characteristics, otherwise, it will be unable to be judged as the car.

As for the other prior art, V. Růžička's research team used deep learning for object detection. Firstly, the low-resolution image of downsampling is used for carrying on the preliminary object detection. In the detected object frame, find out the part of smaller size, in order to be regarded as the detection position required to be focused again. At the nearby position, catch the high-resolution image for carrying on the second object detection. It is necessary to infer that the smaller object can be detected successfully from the first model. The function can be confirmed that the confidence of small object can be obtained. However, if the object is too small, it will be unable to be detected successfully. Under the distant object detection in the driving field, the detection is also unable to be carried out too far.

As for another prior art, M. Najibi's research team used deep learning for carrying on object detection, and the network was added to predict the probable position probability picture of small object. Firstly, the low-resolution image of downsampling is used for carrying on the preliminary object detection, in order to determine the probable position of predicted small object. Then, near the high probability distribution region, catch high-resolution images from the original picture, carry on lesser downsampling, go back to the original procedure and implement it once again repeatedly, until there is no appearance of small objects with high probability. The objects should be large, or small objects appear gradually. If there is appearance of too small objects, and there are no slightly larger objects to guide the system to focus on its region, other small objects might be missed. Furthermore, the numerous inference objects detection models are required. When there are too many objects which are quite dispersed, it will produce too long operation and implementation time. There is another TSENG prior art, that used deep learning object detection technique which had multi-layer convolutional neural network structure. Interweave the pooling layer from each layer, and the object detection layer will be received for processing at the neural network output of rear layers. The results are summarized, and the final object detection result is outputted. This model will focus on smaller object detection in the shallower neural network output part, and the deeper part will focus on larger object detection, which can detect different multi-scale objects. However, due to the shallow layer only catches lesser object characteristics, and it is harder to completely judge the classification of small objects usually, so that the obtained confidence will be relatively poor. As for the detection way of distant small objects, use the low-resolution images to guess the probable position of small objects first, and then catch each position to enter into object detection model for detecting detailed object frame and classification from high-resolution images. As for this detection way, the objects can't be too small, otherwise when using the low-resolution images to predict the probable position of small objects at the first stage, and the characteristics might be too few to judge the object frame and classification, so that the detection might not be successful.

Therefore, it is necessary to improve the prior art of deep learning object detection technique, in order to solve the shortcoming of the prior art of deep learning object detection technique. This is also an urgent topic which has to be improved in the technical field of relevant sectors.

SUMMARY OF THE INVENTION

In view of the above-mentioned description, the present invention proposes an embedded deep learning multi-scale object detection model using real-time distant region locating device and method thereof, in order to solve the shortcoming of the prior art.

The purpose of the present invention is to overcome the shortcoming of existing technique, proposes an embedded multi-function deep learning network, which have good computation efficiency and can develop the optimal distant smaller object detection system without increasing the computation amount.

The purpose of the present invention relates to an embedded deep learning multi-scale object detection model using real-time distant region locating device and method thereof, which catches big objects image through downsampling, and reserves the characteristics of small objects, in order to achieve the purpose of high detection accuracy.

The purpose of the present invention is to base on the above-mentioned embedded deep learning multi-scale object detection model using real-time distant region locating device and method thereof, which also has high-efficiency deep learning object detection structure of distant/nearby object detection accuracy and algorithm operation speed. Under lower computation amounts, it has better object detection accuracy.

The structure of the present invention can be applied to majority of famous deep learning object detection model, and operated in the automotive embedded Nvidia Jetson Xavier platform.

In order to achieve the above-mentioned purpose and other purposes, the present invention provides an embedded deep learning multi-scale object detection model using real-time distant region locating method, which can be used for drive recording to detect the farthest objects in the picture, comprising:

Catching the image from the picture, the image includes the coordinate of original picture;

Making downsampling image processing for the image, to obtain the downsampling image;

Using the multi-function deep learning network to catch the characteristics of downsampling image from the downsampling image;

Inputting the characteristics of the downsampling image to varnish point detection subnetwork, to judge the varnish point;

Regarding the region around varnish point as trim frame, trim the high-resolution image in the trim frame;

Using the multi-function deep learning network to catch the characteristics of high-resolution image from the trimmed high-resolution image;

Using the object detection subnetwork to analyze the object high-resolution result, the object analysis result includes a plurality of object frames; and,

Integrating the varnish point detection result, and the object high-resolution detection results, and outputting the detection results.

After Step (c), the present invention further includes the following steps:

Inputting the characteristics of downsampling image to the object detection subnetwork, analyzing the object downsampling detection results, the object downsampling detection result includes a plurality of downsampling object frames;

Executing the original Step (h), in order to get (h)', integrating the varnish point detection result, the object downsampling detection results, and the object high-resolution detection results, and outputting the detection results.

After Step (d), the present invention further includes Step (h), integrating the varnish point detection result, and the object high-resolution detection results, and outputting the detection results.

In Step (d) of the present invention, the analyzed varnish point detection result is the computed maximum confidence number.

The varnish point detection subnetwork of the present invention is the trained varnish point detection multi-function deep learning network, including 1×1 convolution computation, flat layer and all connecting layers.

Between Step (d) and Step (e), the present invention further comprises:

(d1) Dividing the downsampling images into the coordinate with appointed grids;

(d2) Labelling the varnish point result in the coordinate, to become the varnish point coordinate;

(d3) Converting the varnish point coordinate into the classification;

(d4) Converting the classification into the classification coordinate, and substituting the varnish point coordinate by this classification coordinate.

Step (h) of the present invention includes the followings:

(h1) Making non-maximum restriction processing for these varnish point detection results, object downsampling detection results, and object high-resolution detection results, respectively;

(h2) Using the upper left coordinate of the trim frame as the offset, shift these high-resolution object frames of object high-resolution results to the coordinate of original picture; and,

(h3) Removing these high-resolution object frames at the edge of trimmed high-resolution images, combing the surplus high-resolution object frames and downsampling object frames, and making non-maximum restriction processing.

Step (i) and (d) of the present invention can be executed repeatedly to get the detection results in a plurality of layers.

In order to achieve the above-mentioned purpose and other purposes, the present invention provides an embedded deep learning multi-scale object detection model using real-time distant region locating device, which can be used in drive recording to obtain the farthest objects in the detection picture, comprising:

Image catch unit, used for catching the image of picture; process unit, used for making the downsampling image processing of the image, to obtain the downsampling image, the process unit connects the image catch unit; storage unit, which connects the process unit, the process unit stores the object detection subnetwork, multi-function deep learning network and varnish point detection subnetwork; where, the process unit uses the multi-function deep learning network to catch the characteristics of downsampling image from the downsampling image, inputs the characteristics of downsampling image to the varnish point detection subnetwork, the process unit analyzes the varnish point detection result, to judge the varnish point, the process unit regards the region around varnish point as trim frame, trims the high-resolution image in the trim frame, the process unit uses multi-function deep learning network to catch the characteristics of high-resolution image from the trimmed high-resolution image, uses the object detection subnetwork to analyze the object high-resolution result, the object analysis result includes a plurality of object frames, the process unit integrates the varnish point detection result and the object high-resolution detection results, and outputs the detection results.

The present invention inputs the characteristics of downsampling image to the object detection subnetwork, the process unit analyzes the object downsampling detection results, the object downsampling detection results includes a plurality of downsampling object frames, the process unit integrates the varnish point detection result, the object downsampling detection results, and the object high-resolution detection results, and outputs the detection results.

Compared to the conventional technique, the present invention provides an embedded deep learning multi-scale object detection model using real-time distant region locating method, using the low-resolution input to make nearby object detection at the first stage, and predicts the varnish point region, catches the picture adjacent to varnish point from the high-resolution original picture as the distant picture. Subsequently, carrying on the second object detection, to detection the distant objects, finally, combining the distant/nearby object detection results through a specific post processing algorithm, to obtain a complete object detection result.

Through the image catch device, the present invention uses fixed frequency to catch the image information in front of vehicle, inputs the image with arbitrary resolution, provides clear trichromatic (RGB) image, and higher resolution can be used to detect farther objects.

After the present invention catches the downsampling image to low-resolution image, inputting to the multi-function deep learning network, catching the image characteristics vector matrix, and inputting to object detection subnetwork and varnish point detection subnetwork, to get larger object, that is closer object detection results and image picture varnish point, then based on this varnish point, catch the set small region nearby high-resolution original picture, sent again to the multi-function deep learning network and object detection subnetwork without downsampling, detecting smaller distant object result, finally, using a specific post processing method to combine these two detection results to obtain final result.

Compared to the conventional prior technique, the present invention has the following development advantages:

The present invention adopts the deep learning method to carry on the catch of image characteristics. Compared to the algorithm of conventional technique, the present invention has higher accuracy and stability with respect various weathers, diversified backgrounds, and various object types.

The present invention uses the multi-function deep learning network to compute two tasks of object detection and varnish point detection in a same network, where these two tasks share the trunk structure of network together, which can save the computation amount greatly. Based on this advantage, the present invention can effectively detect distant objects without increasing too much computation amount. Therefore, the present invention can effectively process a great deal of computation amount required by the deep learning network.

As for the sensor required by the present invention, only an image catch device can be adopted to detect nearby and distant objects. Compared to some conventional technique systems, the present invention, the wide-angle lens and telephoto lens should be adopted, in order to detect nearby objects and distant objects, respectively. The present invention can save the more use cost further.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same becomes better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:

FIG. 1 is the flow diagram schematically illustrating the embodiment of an embedded deep learning multi-scale object detection model using real-time distant region locating method for the present invention;

FIG. 2A is the diagram schematically illustrating the catch picture of the present invention;

FIG. 2B is the diagram schematically illustrating the downsampling image of the present invention;

FIG. 2C is the diagram schematically illustrating the varnish point detection result image of the present invention;

FIG. 2D is the diagram schematically illustrating the varnish point detection result image of the present invention diagram;

FIG. 3 is the diagram schematically illustrating the multi-function learning network of the present invention;

FIG. 4A is the flow diagram schematically illustrating the embodiment of the present invention for the steps of analyzing varnish point and the method for trimming the trimmed high-resolution image in the trim frame;

FIG. 4B is the flow diagram schematically illustrating the embodiment of the present invention for integrating the varnish point detection result and object high-resolution detection results, and outputting the detection results;

FIG. 5 is the diagram schematically illustrating the image coordinate of the embodiment of the present invention; and

FIG. 6 is the diagram schematically illustrating the embodiment of an embedded deep learning multi-scale object detection model using real-time distant region locating device for the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENT

The attached figures should be used to describe the implement way of the present invention. In the figures, the same element symbol is used to represent the same element, in order to describe the element more clearly, its size or thickness might be scaled.

Firstly, please refer to FIG. 1 , FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 3 , FIG. 4A, FIG. 4B, FIG. 5 , and FIG. 6 .

FIG. 1 is the flow diagram schematically illustrating the embodiment of an embedded deep learning multi-scale object detection model using real-time distant region locating method for the present invention. It is known that the present invention provides an embedded deep learning multi-scale object detection model using real-time distant region locating method, which can be used for drive recording to detect the farthest objects in the picture, comprising:

Firstly, as shown in Step S100 of FIG. 1 , catch the image V from the picture P, and refer to the catch picture of the present invention shown in FIG. 2A.

As shown in Step S102 of FIG. 1 , make downsampling image processing for the image, to obtain the downsampling image V_(L) as shown in FIG. 6 , and refer to the downsampling image of the present invention shown in FIG. 2B.

As shown in Step S104 of FIG. 1 , use the multi-function deep learning network to catch the characteristics of downsampling image from the downsampling image V_(L), and still refer to the downsampling image of the present invention shown in FIG. 2B.

As shown in Step S106 of FIG. 1 , input the characteristics of the downsampling image V_(L) to varnish point detection subnetwork, analyze varnish point detection result V_(P), to judge the varnish point D, as varnish point detection result image shown in FIG. 2C of the present invention. It has to describe that as the varnish point detection subnetwork diagram shown in FIG. 3 of the present invention. The varnish point detection subnetwork of the present invention is the trained varnish point detection multi-function deep learning network, including 1×1 convolution computation 320, flat layer 340 and all connecting layer 360.

As shown in Step S108 of FIG. 1 , regard the region (that is varnish point detection result V_(P)) around varnish point D shown in FIG. 2C as trim frame, corresponding to the high-resolution image V_(H) of image P, trim the high-resolution image in the trim frame to obtain the high-resolution image V_(FH), as shown in FIG. 2D. It has to describe that the region around varnish point D shown in FIG. 2C is a predetermined distance expanded from the varnish point D as the center, and the predetermined distance can be adjusted in accordance with the image resolution.

Please refer to FIG. 4A of the present invention for the steps of analyzing varnish point, and the steps of trimming the trimmed high-resolution image in the trim frame.

Refer to Step S401 of FIG. 4A, input the characteristics of downsampling image to the varnish point detection subnetwork, analyze varnish point detection result, to judge the varnish point; and as shown in Step S405 of FIG. 4A, regard the region around varnish point as the trim frame, trim the trimmed high-resolution image in the trim frame, further comprising:

Refer again to Step S402 of FIG. 4A, divide the downsampling image V_(L) into the coordinate with appointed grids. For example, divide it into 16×9 grids as shown in the embodiment. It has to describe that the coordinate grids of the present invention can be adjusted in accordance with actual requirement, which is not limited to the grids stated in the above-mentioned embodiment.

Refer again to Step S403 of FIG. 4A, label the varnish point result V_(P) shown in FIG. 2C in the coordinate, to become the varnish point coordinate C_(P), as the image coordinate diagram shown in FIG. 5 .

Refer to Step S404 of FIG. 4A, convert the varnish point coordinate Cp shown in FIG. 5 into the classification.

Refer to Step S405 of FIG. 4A, convert the classification into the classification coordinate, and substitute the varnish point coordinate by this classification coordinate. Where, in Step S401 of FIG. 4A, the analyzed varnish point detection result is the computed maximum confidence number.

As shown in Step S110 of FIG. 1 , the present invention uses the multi-function deep learning network to catch the characteristics of high-resolution image from the trimmed high-resolution image V_(FH), as shown in FIG. 2D.

As shown in Step S112 of FIG. 1 , the object high-resolution detection result is shown by using the object detection subnetwork. The object high-resolution detection results include a plurality of high-resolution object frames, the first high-resolution object frame O_(H1), the second high-resolution object frame O_(H2), the third high-resolution object frame O_(H3), and the fourth high-resolution object frame O_(H4), as shown in FIG. 2D. The object detection subnetwork of the present invention includes the RetinaNet, YOLOv3, FCOS, FoveaBox, RepPoints, Anchor-base, and Anchor-free model.

As shown in Step S114 of FIG. 1 , integrate the varnish point detection result R_(D) shown in FIG. 2C, and the object high-resolution detection results R_(H) shown in FIG. 2D, and output the detection results.

Please refer to FIG. 4B, after Step S104, FIG. 4B further includes the following steps:

As shown in Step S116 of FIG. 1 , input the characteristics of downsampling image to the object detection subnetwork, and analyze the object downsampling detection results R_(L) shown in FIG. 2B. It has to describe that the object downsampling detection results RL of the present invention as shown in FIG. 2B includes a plurality of downsampling object frames, the first downsampling object frame O_(L1) as shown in FIG. 2B, and the second downsampling object frame O_(L2) as shown in FIG. 2B.

As shown in Step S114 of FIG. 1 , integrate the varnish point detection result R_(D) as shown in FIG. 2C, the object downsampling detection results R_(L) as shown in FIG. 2B, and the object high-resolution detection results R_(H) as shown in FIG. 2D, and output the detection results.

It has to describe that the execution of Step S116 in FIG. 1 and Step S106 in FIG. 1 is not limited to one time, Step S116 and Step S106 can be executed repeatedly until a plurality of detection results are obtained.

In another embodiment of the present invention, after Step S106 of FIG. 1 , it further includes Step S114 of FIG. 1 , integrate the varnish point detection result and the object high-resolution detection results, and output the detection results.

On the other hand of the present invention, FIG. 4B is the flow diagram schematically illustrating the embodiment of the present invention for integrating the varnish point detection result and object high-resolution detection results, and outputting the detection results. It is known that in Step S114 of FIG. 1 , the steps for integrating the varnish point detection result and object high-resolution detection results, and outputting the detection results, further comprising:

As shown in Step S421 of FIG. 4B, make non-maximum restriction processing for these varnish point detection result, object downsampling detection results, and object high-resolution detection results, respectively.

As shown in Step S422 of FIG. 4B, use the upper left coordinate of the trim frame as the offset, shift these high-resolution object frames of object high-resolution results to the coordinate of original picture.

As shown in Step S423 of FIG. 4B, remove these high-resolution object frames at the edge of trimmed high-resolution images, combing the surplus high-resolution object frames and downsampling object frames, and making non-maximum restriction processing.

In the present invention, Step S116 of FIG. 1 to Step S114 of FIG. 1 , or Step S106 of FIG. 1 to Step S114 of FIG. 1 to FIG. 1 is the first stage. Step S106 of FIG. 1 to Step S108, Step S110, Step S112, and Step S114 is the second stage. The present invention is implemented in two stages. The first stage can be implemented repeatedly, carry on the focus of more multi-layer visual field. Finally, carry on the varnish point detection process specified in the second stage. But the present invention is not only limited to the object detection, any multi-scale image identification task is also the scope of the present invention.

FIG. 6 is the diagram schematically illustrating the embodiment of an embedded deep learning multi-scale object detection model using real-time distant region locating device for the present invention. Please refer to the above-mentioned FIG. 2A, FIG. 2B, FIG. 2C, and FIG. 2D. The embedded deep learning multi-scale object detection model using real-time distant region locating device 600 can be used in the drive recording to detect the farthest objects in the picture, comprising: image catch unit 610, the image catch unit 610 is used to catch the image V of picture P (refer to FIG. 2A), process unit 620, and storage unit 630.

As the process unit 620 shown in FIG. 6 , make downsampling image V processing for the image in FIG. 2A, to obtain the downsampling image V_(L). The process unit 620 connects the image catch unit 610 and storage unit 630. The process unit 630 stores the object detection subnetwork 632, multi-function deep learning network 634, and varnish point detection subnetwork 636. Wherein the process unit 620 uses the multi-function deep learning network 634 to catch the characteristics of downsampling image from downsampling image V_(L), as shown in FIG. 2B.

As for the structure of varnish point detection subnetwork 636 shown in FIG. 6 , FIG. 3 shows the diagram schematically illustrating the multi-function learning network of the present invention. The output matrix (as shown in FIG. 3 ) shown in the characteristics of downsampling image 310 caught by the multi-function deep learning network 634 shown in FIG. 6 can be sent to the deep learning network of trained varnish point detection, its composition is 1×1 convolution computation 320, flat layer 340 and all connecting layer 360, first vector matrix 330, second vector matrix 350, and finally outputted third vector matrix 370, in order to compute its maximum confidence number, that is the predicted result R_(D) of varnish point shown in FIG. 2C.

Inputting the characteristics of downsampling image is shown in FIG. 6 to the varnish point detection subnetwork 632, process unit 620 analyzes the varnish point detection result R_(D) shown in FIG. 2C, judge the varnish point D shown in FIG. 2C. As the process unit 620 shown in FIG. 6 , regard the region (varnish point detection result V_(P) shown in FIG. 2C) around varnish point as trim frame, trim the high-resolution image V_(FH) in the trim frame as shown in FIG. 2D.

The process unit 620 of FIG. 6 uses the multi-function deep learning network 634 to catch the characteristics of high-resolution image from the trimmed high-resolution image V_(H) shown in FIG. 2D, and uses the object detection subnetwork 632 to analyze the object high-resolution detection results R_(H) shown in FIG. 2D (also as shown in FIG. 6 ). As shown in FIG. 2D, the object high-resolution detection results R_(H) includes a plurality of high-resolution object frames, the first high-resolution object frame O_(H1), the second high-resolution object frame OH₂, the third high-resolution object frame OH₃, and the fourth high-resolution object frame O_(H4).

The object detection subnetwork 632 shown in FIG. 6 is RetinaNet, YOLOv3, FCOS, FoveaBox, RepPoints, Anchor-base, and Anchor-free model. The process unit 620 shown in FIG. 6 integrates the varnish point detection result R_(D) shown in FIG. 2C, and the object high-resolution detection results R_(H) shown in FIG. 2D, and output the detection result R shown in FIG. 6 .

The process unit 620 shown in FIG. 6 inputs the characteristics of downsampling image to the object detection subnetwork 632, and analyzes the object downsampling detection results. The object downsampling detection results R_(L) shown in FIG. 6 includes a plurality of downsampling object frames, the first downsampling object frame O_(L1) shown in FIG. 2B, and the second downsampling object frame O_(L2) shown in FIG. 2B.

The process unit 620 shown in FIG. 6 integrates the varnish point detection result R_(D) shown in FIG. 2C, the object downsampling detection results R_(L) shown in FIG. 2B, and the object high-resolution detection results R_(H), and outputs the detection results.

The present invention relates to an object detection method, which can be used for the visual angle of forward drive recorder, detects the farthest place of picture automatically, and focuses on the distant object. It is trying to solve the problem of advanced driver assistance system, where the detection distance is often not far enough. The present invention has an efficient multi-scale object detection network (ConcentrateNet), which can search the varnish point of picture automatically, and pay close attention to its nearby region. At the first inference in the model, it will produce the objects under the larger visual field for the prediction of object detection result and varnish point position. The varnish point position represents the farthest place of picture. Then use the varnish point position to make model inference around nearby region once again, to obtain the detection result of distant object. Finally, use non-maximum restriction processing to combine these two results.

The structure of the present invention can be applied to most object detection models. It is able to use several advanced object detection models and add the multi-scale object detection network structure to carry on the test. Compared to the original model using higher input picture to draw higher quality, the less computation amount can be used to obtain higher accuracy. For example, under the information BDD100K in large vehicle, YOLOv3 can directly use 960×540 high resolution to get AP 28.2% accuracy. The multi-scale object detection network uses 640×360 low-resolution still can get AP 30.7% accuracy. The recall rate of small object even can be increased from AR 24.9% to 35.7%. In addition, the present invention can test the multi-scale object detection network through the low power consumption embedded NVIDIA Jetson AGX Xavier platform.

The present invention provides an embedded deep learning multi-scale object detection model using real-time distant region locating method, which can be used for drive recording to detect the farthest objects in the picture, comprising:

(a) Catching the image from the picture, the image includes the coordinate of original picture;

(b) Making downsampling image processing for the image, to obtain the downsampling image;

(c) Using the multi-function deep learning network to catch the characteristics of downsampling image from the downsampling image;

(d) Inputting the characteristics of the downsampling image to varnish point detection subnetwork, to judge the varnish point;

(e) Regarding the region around varnish point as trim frame, trim the high-resolution image in the trim frame;

(f) Using the multi-function deep learning network to catch the characteristics of high-resolution image from the trimmed high-resolution image;

(g) Using the object detection subnetwork to analyze the object high-resolution result, the object analysis result includes a plurality of object frames; and,

(h) Integrating the varnish point detection result, and the object high-resolution detection results, and outputting the detection results.

After Step (c), the present invention further includes the following steps:

(i) Inputting the characteristics of downsampling image to the object detection subnetwork, analyzing the object downsampling detection results, the object downsampling detection result includes a plurality of downsampling object frames;

Executing the original Step (h), in order to get (h)', integrating the varnish point detection result, the object downsampling detection results, and the object high-resolution detection results, and outputting the detection results.

After Step (d), the present invention further includes Step (h), integrating the varnish point detection result, and the object high-resolution detection results, and outputting the detection results.

In Step (d) of the present invention, the analyzed varnish point detection result is the computed maximum confidence number.

The varnish point detection subnetwork of the present invention is the trained varnish point detection multi-function deep learning network, including 1×1 convolution computation, flat layer and all connecting layer.

Between Step (d) and Step (e), the present invention further comprises:

(d1) Dividing the downsampling images into the coordinate with appointed grids;

(d2) Labelling the varnish point result in the coordinate, to become the varnish point coordinate;

(d3) Converting the varnish point coordinate into the classification;

(d4) Converting the classification into the classification coordinate, and substituting the varnish point coordinate by this classification coordinate.

Step (h) of the present invention includes:

(h1) Making non-maximum restriction processing for these varnish point detection results, object downsampling detection results, and object high-resolution detection results, respectively;

(h2) Using the upper left coordinate of the trim frame as the offset, shift these high-resolution object frames of object high-resolution results to the coordinate of original picture; and,

(h3) Removing these high-resolution object frames at the edge of trimmed high-resolution images, combing the surplus high-resolution object frames and downsampling object frames, and making non-maximum restriction processing.

Step (i) and (d) of the present invention can be executed repeatedly to get the detection results in a plurality of layers.

The object detection subnetwork of the present invention includes the RetinaNet, YOLOv3, FCOS, FoveaBox, RepPoints, Anchor-base, and Anchor-free model.

It is understood that various other modifications will be apparent to and can be readily made by those skilled in the art without departing from the scope and spirit of this invention. Accordingly, it is not intended that the scope of the claims appended hereto be limited to the description as set forth herein, but rather that the claims be construed as encompassing all the features of patentable novelty that reside in the present invention, including all features that would be treated as equivalents thereof by those skilled in the art to which this invention pertains. 

What is claimed is:
 1. An embedded deep learning multi-scale object detection model using real-time distant region locating method for drive recording to detect far objects in a picture, comprising: (a) catching an image from a picture, said image having a coordinate of original picture; (b) making a downsampling image processing for said image to obtain said downsampling image; (c) using a multi-function deep learning network to catch characteristics of said downsampling image from said downsampling image; (d) inputting said characteristics of said downsampling image to varnish a point detection subnetwork to judge a varnish point; (e) regarding a region around a varnish point as a trim frame, to trim a high-resolution image in said trim frame; (f) using a multi-function deep learning network to catch said characteristics of said high-resolution image from a trimmed high-resolution image; (g) using an object detection subnetwork to analyze an object high-resolution result, an object analysis result comprising a plurality of object frames; and (h) integrating a varnish point detection result, and said object high-resolution detection results, and outputting detection results.
 2. The method according to claim 1, wherein after Step (c), further comprising: (i) inputting said characteristics of downsampling image to an object detection subnetwork, analyzing said object downsampling detection results, said object downsampling detection result comprising a plurality of downsampling object frames; and executing said original Step (h), in order to obtain (h)', integrating said varnish point detection result, said object downsampling detection results, and said object high-resolution detection results, and outputting detection results.
 3. The method according to claim 2, wherein after Step (d), further comprises Step (h), integrating said varnish point detection result, and said object high-resolution detection results, and outputting detection results.
 4. The method according to claim 1, wherein in Step (d), said analyzed varnish point detection result comprises a computed maximum confidence number.
 5. The method according to claim 1, wherein said varnish point detection subnetwork comprises a trained varnish point detection multi-function deep learning network, including a 1×1 convolution computation, a flat layer and all connecting layer.
 6. The method according to claim 2, wherein between Step (d) and Step (e), further comprises: (d1) dividing downsampling images into a coordinate with appointed grids; (d2) labeling a varnish point result in said coordinate, to become a varnish point coordinate; (d3) converting said varnish point coordinate into a classification; and (d4) converting said classification into a classification coordinate, and substituting said varnish point coordinate by said classification coordinate.
 7. The method according to claim 2, wherein Step (h) further includes: (h1) making non-maximum restriction processing for a varnish point detection result, objecting downsampling detection results, and objecting high-resolution detection results, respectively; (h2) using an upper left coordinate of a trim frame as an offset, shifting high-resolution object frames of object high-resolution results to a coordinate of original picture; and (h3) removing high-resolution object frames at an edge of trimmed high-resolution images, combing surplus high-resolution object frames and downsampling object frames, and making a non-maximum restriction processing.
 8. The method according to claim 1, wherein the object detection subnetwork is selected from the group of RetinaNet, YOLOv3, FCOS, FoveaBox, RepPoints, Anchor-base, and Anchor-free model.
 9. The method according to claim 1, wherein Step (i) and (d) comprises been executed repeatedly to obtain detection results in a plurality of layers.
 10. An embedded deep learning multi-scale object detection model using real-time distant region locating device, used in driving recording to obtain farthest objects a detection picture, comprising: an image catch unit, used for catching an image of picture; a process unit, used for making a downsampling image processing of the image, to obtain a downsampling image, said process unit connecting said image catch unit; and a storage unit, connecting said process unit, said process unit storing an object detection subnetwork, a multi-function deep learning network and a varnish point detection subnetwork; wherein, the process unit uses said multi-function deep learning network to catch a characteristic of said downsampling image from said downsampling image, inputting said characteristics of said downsampling image to said varnish point detection subnetwork, said process unit analyzing varnish point detection result, to judge a varnish point; said process unit regarding a region around varnish point as a trim frame, to trim a high-resolution image in said trim frame; said process unit using said multi-function deep learning network to catch said characteristics of said high-resolution image from a trimmed high-resolution image, using said object detection subnetwork to analyze an object high-resolution result, an object analysis result having a plurality of object frames, a process unit integrating a varnish point detection result and object high-resolution detection results, and outputting detection results.
 11. The method according to claim 10, providing an embedded deep learning multi-scale object detection model using real-time distant region locating device, wherein, inputting a characteristic of downsampling image to an object detection subnetwork, a process unit analyzing object downsampling detection results, object downsampling detection results comprises a plurality of downsampling object frames, a process unit integrating a varnish point detection result, said object downsampling detection results, and object high-resolution detection results, and outputting detection results.
 12. The method according to claim 10, providing an embedded deep learning multi-scale object detection model using real-time distant region locating device, wherein, an object detection subnetwork is selected from the group of RetinaNet, YOLOv3, FCOS, FoveaBox, RepPoints, Anchor-base, and Anchor-free model. 